Import modules

import numpy as np
import pandas as pd
import plotly.express as px
import matplotlib.pyplot as plt
import plotly.graph_objects as go
import plotly.figure_factory as ff

from sklearn import metrics
from sklearn.cluster import KMeans
from sklearn.mixture import GaussianMixture

pd.set_option('display.max_columns', None)

Initialize and load dataset

raw_df = pd.read_csv("../datasets/EDA_df.csv")
raw_df.head(10)

	customer_id	gender	senior_citizen	partner	dependents	phone_service	multiple_lines	online_security	online_backup	device_protection	tech_support	streaming_tv	streaming_movies	paperless_billing	internet_service_DSL	internet_service_fiber_optic	no_internet_service	contract_month_to_month	contract_one_year	contract_two_year	payment_method_bank_transfer_automatic	payment_method_credit_card_automatic	payment_method_electronic_check	payment_method_mailed_check	tenure	monthly_charges	total_charges	churn
0	7590-VHVEG	0	0	0	0	0	0	0	1	0	0	0	0	1	1.0	0.0	0.0	1.0	0.0	0.0	0.0	0.0	1.0	0.0	1	29.85	29.85	0
1	5575-GNVDE	1	0	0	0	0	0	1	0	1	0	0	0	0	1.0	0.0	0.0	0.0	1.0	0.0	0.0	0.0	0.0	1.0	34	56.95	1936.30	0
2	3668-QPYBK	1	0	0	0	0	0	1	1	0	0	0	0	1	1.0	0.0	0.0	1.0	0.0	0.0	0.0	0.0	0.0	1.0	2	53.85	107.70	1
3	7795-CFOCW	1	0	0	0	0	0	1	0	1	1	0	0	0	1.0	0.0	0.0	0.0	1.0	0.0	1.0	0.0	0.0	0.0	45	42.30	1903.50	0
4	9237-HQITU	0	0	0	0	0	0	0	0	0	0	0	0	1	0.0	1.0	0.0	1.0	0.0	0.0	0.0	0.0	1.0	0.0	2	70.70	141.40	1
5	9305-CDSKC	0	0	0	0	0	0	0	0	1	0	1	1	1	0.0	1.0	0.0	1.0	0.0	0.0	0.0	0.0	1.0	0.0	8	99.65	797.20	1
6	1452-KIOVK	1	0	0	0	0	0	0	1	0	0	1	0	1	0.0	1.0	0.0	1.0	0.0	0.0	0.0	1.0	0.0	0.0	22	89.10	1960.20	0
7	6713-OKOMC	0	0	0	0	0	0	1	0	0	0	0	0	0	1.0	0.0	0.0	1.0	0.0	0.0	0.0	0.0	0.0	1.0	10	29.75	297.50	0
8	7892-POOKP	0	0	0	0	0	0	0	0	1	1	1	1	1	0.0	1.0	0.0	1.0	0.0	0.0	0.0	0.0	1.0	0.0	28	104.80	2934.40	1
9	6388-TABGU	1	0	0	0	0	0	1	1	0	0	0	0	0	1.0	0.0	0.0	0.0	1.0	0.0	1.0	0.0	0.0	0.0	62	56.15	3481.30	0

KMeans clustering with 2 features

Select 2 feature from dataset

X_cluster_2_features = raw_df.loc[:, ['tenure', 'monthly_charges']]

Plotting feature with scatter plot

fig = px.scatter(X_cluster_2_features, x="tenure", y="monthly_charges", color=raw_df['churn'], size=raw_df['total_charges'])

fig.update_layout(
    title       = 'Tenure and Monthly Charges',
    title_x     = 0.5,
    xaxis_title = 'Tenure',
    yaxis_title = 'Monthly Charges'
)

fig.show()

KMeans Clustering without WCSS evaluation

Centroid yang dipilih adalah 4, karena hasil cluster bisa dijelaskan menggunakan visualisasi.

Initialize model

kmeans_4_centroid = KMeans(n_clusters=4, random_state=42)

Train model and predict cluster

X_cluster_2_features['cluster_pred'] = kmeans_4_centroid.fit_predict(X_cluster_2_features)

Show cluster prediction

cluster_pred_4_centroid = X_cluster_2_features
cluster_pred_4_centroid.head(10)

	tenure	monthly_charges	cluster_pred
0	1	29.85	3
1	34	56.95	0
2	2	53.85	3
3	45	42.30	0
4	2	70.70	2
5	8	99.65	2
6	22	89.10	2
7	10	29.75	3
8	28	104.80	2
9	62	56.15	0

Visualize with scatterplot

fig = px.scatter(cluster_pred_4_centroid, x="tenure", y="monthly_charges", color="cluster_pred", size=raw_df['total_charges'])
fig.show()

1. Cluster menggunakan 4 centroid.
2. Terdapat empat cluster yang dapat dikelompokkan berdasarkan beban tagihan pengguna dalam jangka waktu tertentu.
3. Empat cluster itu, antara lain:

• Cluster A(ungu): tenure > 32 bulan dan monthly_charges > 60 dollars.
• Cluster B(orange): tenure <= 32 bulan dan monthly_charges > 60 dollars.
• Cluster C(biru): tenure > 32 bulan dan monthly_charges <= 60 dollars.
• Cluster D(kuning): tenure <= 32 bulan dan monthly_charges <= 60 dollars.

KMeans Clustering with more than 2 variables and WCSS evaluation

Menggunakan 27 fitur.

Determine cluster with WCSS evaluation

wcss = []

for i in range(1, 11):
    kmeans = KMeans(i, random_state=42)
    kmeans.fit(raw_df.iloc[:, 1:28])
    wcss.append(kmeans.inertia_)

Plotting WCSS

fig = go.Figure(data=go.Scatter(x=np.arange(1, 11), y=wcss))

fig.update_layout(
    title       = 'Within-Cluster Sum of Squares',
    title_x     = 0.5,
    xaxis_title = 'Number of Clusters',
    yaxis_title = 'Score'
)

fig.show()

• Semakin bertambah jumlah cluster, nilai inertia semakin menurun.
• Jumlah cluster diatas 5, penurunan nilai inertia awal dan akhir tidak signifikan.
• Dari grafik WCSS di atas, dapat dilihat bahwa cluster yang memiliki nilai penurunan inertia diawal dan akhir secara signifikan adalah cluster dengan jumlah 2 dan 3.
• Jumlah cluster yang dipilih adalah 2, karena inertia score paling besar dibanding cluster yang lain, yaitu 7918484236

Initialize model

kmeans_2_centroid = KMeans(n_clusters=2, random_state=42)

Add cluster predict to raw dataframe

raw_df['cluster_pred'] = kmeans_2_centroid.fit_predict(raw_df.iloc[:, 1:28])
cluster_pred_df        = raw_df

Visualize cluster predict in 2 dimension

fig = px.scatter(cluster_pred_df, x="tenure", y="monthly_charges", color="cluster_pred", size='total_charges')
fig.show()

Visualize cluster predict in 3 dimension

fig = px.scatter_3d(cluster_pred_df, x='tenure', y='monthly_charges', z='total_charges', color='cluster_pred')

fig.show()

• Cluster dibagi berdasarkan total charges, yaitu Cluster Low-Mid dengan 0 dollars < total charges <= 3000 dollars dan Cluster Mid-High dengan 3000 dollars < total charges <= 8500 dollars.

Anomaly Detection with Gaussian Mixture Model

Without oversampling dataset

Initialize dataset

without_oversampling_df = pd.read_csv('../datasets/without-oversampling/EDA_df.csv')
without_oversampling_df.shape

(7043, 28)

Determine number of cluster

• Jumlah cluster ditentukan dengan menggunakan BIC (Bayesian information criterion) dan AIC (Akaike information criterion).
• Score evaluasi antara BIC dan AIC dibandingkan untuk setiap jumlah komponen.
• Score merupakan nilai dari information criterion.
• Makin kecil nilai information criterion dan gap antara AIC dan BIC kecil, maka cluster tersebut memiliki nilai criterion minimum.
• BIC cenderung menghasilkan model yang simpel, dibanding AIC.

aic_scores = []
bic_scores = []

for i in range(1, 21):
    gm = GaussianMixture(n_components=i, random_state=42).fit(without_oversampling_df.iloc[:, 1:28])
    aic_scores.append(gm.aic(without_oversampling_df.iloc[:, 1:28]))
    bic_scores.append(gm.bic(without_oversampling_df.iloc[:, 1:28]))

fig = go.Figure()

fig.add_trace(
    go.Scatter(
        x    = np.arange(1, 21),
        y    = aic_scores,
        mode = 'lines+markers',
        name = 'AIC'
    )
)

fig.add_trace(
    go.Scatter(
        x    = np.arange(1, 21),
        y    = bic_scores,
        mode = 'lines+markers',
        name = 'BIC'
    )
)

fig.update_layout(
    title       = 'Cluster Selection with AIC and BIC',
    title_x     = 0.5,
    xaxis_title = 'Number of Clusters',
    yaxis_title = 'Score'
)

fig.show()

• Berdasarkan hasil visualisasi nilai cluster dengan information criterion terendah berada pada cluster benilai 19.
• Karena jumlah cluster 19 terlalu banyak dan sulit untuk dijelaskan, maka dipilih cluster dengan jumlah 2. Cluster dengan nilai 2, gap antara AIC dan BIC score sangat kecil, walaupun nilai score secara general bukan yang terkecil.

Select feature and train model

X_without_oversampling            = without_oversampling_df.iloc[:, 1:28]
gm_pred                           = GaussianMixture(n_components=2, random_state=42).fit_predict(X_without_oversampling)
X_without_oversampling['gm_pred'] = gm_pred

X_without_oversampling.head(10)

	gender	senior_citizen	partner	dependents	phone_service	multiple_lines	online_security	online_backup	device_protection	tech_support	streaming_tv	streaming_movies	paperless_billing	internet_service_DSL	internet_service_fiber_optic	no_internet_service	contract_month_to_month	contract_one_year	contract_two_year	payment_method_bank_transfer_automatic	payment_method_credit_card_automatic	payment_method_electronic_check	payment_method_mailed_check	tenure	monthly_charges	total_charges	churn	gm_pred
0	0	0	0	0	0	0	0	1	0	0	0	0	1	1.0	0.0	0.0	1.0	0.0	0.0	0.0	0.0	1.0	0.0	1	29.85	29.85	0	1
1	1	0	0	0	0	0	1	0	1	0	0	0	0	1.0	0.0	0.0	0.0	1.0	0.0	0.0	0.0	0.0	1.0	34	56.95	1936.30	0	1
2	1	0	0	0	0	0	1	1	0	0	0	0	1	1.0	0.0	0.0	1.0	0.0	0.0	0.0	0.0	0.0	1.0	2	53.85	107.70	1	1
3	1	0	0	0	0	0	1	0	1	1	0	0	0	1.0	0.0	0.0	0.0	1.0	0.0	1.0	0.0	0.0	0.0	45	42.30	1903.50	0	1
4	0	0	0	0	0	0	0	0	0	0	0	0	1	0.0	1.0	0.0	1.0	0.0	0.0	0.0	0.0	1.0	0.0	2	70.70	141.40	1	1
5	0	0	0	0	0	0	0	0	1	0	1	1	1	0.0	1.0	0.0	1.0	0.0	0.0	0.0	0.0	1.0	0.0	8	99.65	797.20	1	1
6	1	0	0	0	0	0	0	1	0	0	1	0	1	0.0	1.0	0.0	1.0	0.0	0.0	0.0	1.0	0.0	0.0	22	89.10	1960.20	0	1
7	0	0	0	0	0	0	1	0	0	0	0	0	0	1.0	0.0	0.0	1.0	0.0	0.0	0.0	0.0	0.0	1.0	10	29.75	297.50	0	1
8	0	0	0	0	0	0	0	0	1	1	1	1	1	0.0	1.0	0.0	1.0	0.0	0.0	0.0	0.0	1.0	0.0	28	104.80	2934.40	1	1
9	1	0	0	0	0	0	1	1	0	0	0	0	0	1.0	0.0	0.0	0.0	1.0	0.0	1.0	0.0	0.0	0.0	62	56.15	3481.30	0	1

Visualize Gaussian Mixture Model predict

fig = px.scatter(X_without_oversampling, x="tenure", y="monthly_charges", color="gm_pred", size='total_charges')
fig.show()

Visualize Gaussian Mixture Model predict in 3 dimension

fig = px.scatter_3d(X_without_oversampling, x='tenure', y='monthly_charges', z='total_charges', color='gm_pred')

fig.show()

print(f"Number of anomaly datapoint: {X_without_oversampling[(X_without_oversampling['monthly_charges'] >= 40) & (X_without_oversampling['monthly_charges'] <= 60) & (X_without_oversampling['gm_pred'] == 0)].shape[0]}")

Number of anomaly datapoint: 207

• Gaussian Mixture Model dengan jumlah cluster 2, yaitu cluster A(biru) dan B(kuning), terdapat anomali pada cluster A(biru).
• Pada cluster A(biru) terdapat beberapa data poin yang berada di cluster B(kuning), dimana 40 < monthly charges < 60 dollars. Data poin ini dikategorikan sebagai anomali.
• Jumlah datapoin yang anomali sebanyak 207.

fig = px.scatter_3d(X_without_oversampling, x='tenure', y='monthly_charges', z='total_charges', color='churn')

fig.show()

With oversampling dataset

Initialize dataset

oversampling_df = pd.read_csv('../datasets/oversampling/EDA_telco_customer_churn.csv')
oversampling_df.shape

(10348, 28)

Determine number of cluster

X_oversampling  = oversampling_df.iloc[:, 1:28]
aic_scores      = []
bic_scores      = []

for i in range(1, 21):
    gm = GaussianMixture(n_components=i, random_state=42).fit(X_oversampling)
    aic_scores.append(gm.aic(without_oversampling_df.iloc[:, 1:28]))
    bic_scores.append(gm.bic(without_oversampling_df.iloc[:, 1:28]))

fig = go.Figure()

fig.add_trace(
    go.Scatter(
        x    = np.arange(1, 21),
        y    = aic_scores,
        mode = 'lines+markers',
        name = 'AIC'
    )
)

fig.add_trace(
    go.Scatter(
        x    = np.arange(1, 21),
        y    = bic_scores,
        mode = 'lines+markers',
        name = 'BIC'
    )
)

fig.update_layout(
    title       = 'Cluster Selection with AIC and BIC',
    title_x     = 0.5,
    xaxis_title = 'Number of Clusters',
    yaxis_title = 'Score'
)

fig.show()

gm_pred                      = GaussianMixture(n_components=2, random_state=42).fit_predict(X_oversampling)
X_oversampling['gm_pred'] = gm_pred

X_oversampling.head(10)

	gender	senior_citizen	partner	dependents	phone_service	multiple_lines	online_security	online_backup	device_protection	tech_support	streaming_tv	streaming_movies	paperless_billing	internet_service_DSL	internet_service_fiber_optic	no_internet_service	contract_month_to_month	contract_one_year	contract_two_year	payment_method_bank_transfer_automatic	payment_method_credit_card_automatic	payment_method_electronic_check	payment_method_mailed_check	tenure	monthly_charges	total_charges	churn	gm_pred
0	0	0	0	0	0	0	0	1	0	0	0	0	1	1.0	0.0	0.0	1.0	0.0	0.0	0.0	0.0	1.0	0.0	1	29.85	29.85	0	0
1	1	0	0	0	0	0	1	0	1	0	0	0	0	1.0	0.0	0.0	0.0	1.0	0.0	0.0	0.0	0.0	1.0	34	56.95	1936.30	0	1
2	1	0	0	0	0	0	1	1	0	0	0	0	1	1.0	0.0	0.0	1.0	0.0	0.0	0.0	0.0	0.0	1.0	2	53.85	107.70	1	1
3	1	0	0	0	0	0	1	0	1	1	0	0	0	1.0	0.0	0.0	0.0	1.0	0.0	1.0	0.0	0.0	0.0	45	42.30	1903.50	0	1
4	0	0	0	0	0	0	0	0	0	0	0	0	1	0.0	1.0	0.0	1.0	0.0	0.0	0.0	0.0	1.0	0.0	2	70.70	141.40	1	1
5	0	0	0	0	0	0	0	0	1	0	1	1	1	0.0	1.0	0.0	1.0	0.0	0.0	0.0	0.0	1.0	0.0	8	99.65	797.20	1	1
6	1	0	0	0	0	0	0	1	0	0	1	0	1	0.0	1.0	0.0	1.0	0.0	0.0	0.0	1.0	0.0	0.0	22	89.10	1960.20	0	1
7	0	0	0	0	0	0	1	0	0	0	0	0	0	1.0	0.0	0.0	1.0	0.0	0.0	0.0	0.0	0.0	1.0	10	29.75	297.50	0	1
8	0	0	0	0	0	0	0	0	1	1	1	1	1	0.0	1.0	0.0	1.0	0.0	0.0	0.0	0.0	1.0	0.0	28	104.80	2934.40	1	1
9	1	0	0	0	0	0	1	1	0	0	0	0	0	1.0	0.0	0.0	0.0	1.0	0.0	1.0	0.0	0.0	0.0	62	56.15	3481.30	0	1

Visualize Gaussian Mixture Model predict

fig = px.scatter(X_oversampling, x="tenure", y="monthly_charges", color="gm_pred", size='total_charges')
fig.show()

fig = px.scatter_3d(X_oversampling, x='tenure', y='monthly_charges', z='total_charges', color='gm_pred')

fig.show()

print(f"Number of anomaly datapoint: {X_oversampling[(X_oversampling['monthly_charges'] >= 40) & (X_oversampling['monthly_charges'] <= 60) & (X_oversampling['gm_pred'] == 0)].shape[0]}")

Number of anomaly datapoint: 426

Jumlah anomali yang terdeteksi bertambah dengan menggunakan dataset oversampling. Anomali terjadi pada datapoin dengan monthly charges berada direntang 40-60 dollars.

Check outlier

Untuk mengecek kepastian anomali, kami mengecek distribusi tenure, monthly charges, dan total charges menggunakan box plot dan distribution plot.

Tenure

• Tidak ada outlier.
• Sebaran data tidak terdistribusi normal.
• Skewed right.

fig = px.box(oversampling_df, x="tenure")
fig.show()

hist_data = [oversampling_df['tenure']]
group_labels = ['tenure']

fig = ff.create_distplot(hist_data, group_labels)
fig.show()

Monthly Charges

• Tidak ada outlier.
• Sebaran data tidak terdistribusi normal.
• Skewed right.

fig = px.box(oversampling_df, x="monthly_charges")
fig.show()

hist_data = [oversampling_df['monthly_charges']]
group_labels = ['monthly charges']

fig = ff.create_distplot(hist_data, group_labels)
fig.show()

Total charges

• Terdapat outlier.
• Sebaran data tidak terdistribusi normal.
• Skewed right.

fig = px.box(oversampling_df, x="total_charges")
fig.show()

hist_data = [oversampling_df['total_charges']]
group_labels = ['total charges']

fig = ff.create_distplot(hist_data, group_labels)
fig.show()

Reference

• Scikit Learn - Silhouette Coefficient https://scikit-learn.org/stable/modules/clustering.html#silhouette-coefficient
• Gaussian Mixture Model clustering: how to select the number of components (clusters) - Vincenzo Lavorini https://towardsdatascience.com/gaussian-mixture-model-clusterization-how-to-select-the-number-of-components-clusters-553bef45f6e4